Recently, a digital camcorder or digital still camera has come to have an imager, including photosensing elements at tremendously high densities, and have had their definitions increased by leaps and bounds. Generally speaking, the higher the resolution of an image captured, the more finely the user can record the scene before him or her. That is why there is an increasing demand for such high-resolution shooting.
The resolutions are classifiable into the spatial resolution and the temporal resolution. The resolution of the former type corresponds to the number of pixels that make up a single picture. If there are two imaging sensors of the same size but with different number of pixels, then an imaging sensor with the greater number of pixels will have a higher spatial resolution, and can record the details of the subject more finely, than the other imaging sensor. As a result, with such an imaging sensor, the user can shoot a still picture with higher presence and higher image quality. On the other hand, the resolution of the latter type corresponds to the number of pictures shot per unit time. An imaging sensor that can shoot a greater number of pictures per second will have a higher temporal resolution. Thus, with such an imaging sensor, the user can track even quick motions of the subject precisely, and can shoot a moving picture with smoother motion.
However, if the user wants to shoot the scene before him or her as finely as possible, he or she often opts for the highest possible spatial and temporal resolutions, thus making the data size of the picture huge. A DV camcorder, for example, will produce an image with a size of 720 pixels×480 lines. Supposing the frame rate is 30 frames per second, a 24 bit RGB color signal should have a data transfer rate of 249 Mbps (megabits per second)(=720 pixels×480 lines×24 bits×30). Meanwhile, a high definition camcorder will produce an image with a size of 1,920 pixels×1,080 lines. And the data transfer rate will be 1.5 Gbps (=1,920 pixels×1,080 lines×24 bits×30), which is six times as high as that of the DV camcorder. If the temporal resolution was doubled and the frame rate was increased to 60 frames per second, then the data transfer rate would be 3 Gbps. In such a situation, the data transfer rate should always be that high since the data has been output from the imaging sensor of the camera and until it is written on a storage medium or until the display monitor is scanned. However, with that high data transfer rate, the load on a normal consumer camcorder or digital camera would be too heavy to handle. That is why by utilizing its redundancy, the data is usually compressed to keep the device as small as possible, cut down the power dissipation, and minimize the cost. Even in camcorders or digital cameras for business use, the data size is also often cut down by compression. And only expensive editing systems for professional use will operate at the original data transfer rate.
As can be seen, to transmit and receive such a high-definition image, of which the spatial and temporal resolutions are both high, at as low a data transfer rate as possible, it is important to acquire only essential data and cut down the redundancy. For that purpose, a technique for acquiring only essential data when an image is captured would work fine. For example, Patent Document No. 1 discloses a technique for acquiring image data 1 with high spatial resolution but low temporal resolution and image data 2 with low spatial resolution but high temporal resolution as shown in FIG. 28(a) and then generating image data 3 with high spatial resolution and high temporal resolution by performing image processing on them as shown in FIG. 28(b).
Supposing the shooting time is one second, the image data 1 has an image size of 1,920 pixels×1,080 lines, and the image data 2 has an image size of 640 pixels×360 lines in the example illustrated in FIG. 28, the 8-bit luminance signal shown in FIG. 28(a) comes to have a data transfer rate of 54 Mbps (=1,920 pixels×1,080 lines×8 bits×2+640 pixels×320 lines×8 bits×13). In FIG. 28(b), on the other hand, as there are eight pictures each consisting of 1,920 pixels×1,080 lines, the 8-bit luminance signal comes to have a data transfer rate of 133 Mbps (==1,920 pixels×1,080 lines×8 bits×8). Consequently, the data transfer rate in FIG. 28(b) is approximately 2.4 times as high as the one in FIG. 28(a).
FIG. 29 is a flowchart showing the procedure for generating the image data 3. As shown in FIG. 29, a motion vector is estimated with the low-spatial-resolution image represented by the image data 2. Based on a result of the motion estimation, the high-spatial-resolution image of the image data 1 is subjected to motion compensation. Next, the DCT (discrete cosine transform) spectrum of the motion-compensated high-spatial-resolution image is synthesized with that of its associated low-spatial-resolution image, and then an inverse DCT is carried out on the synthesized spectrum, thereby obtaining a target high-spatial-resolution, high-temporal-resolution image.
FIG. 30 illustrates an exemplary configuration for a composite sensor camera 4 for acquiring the image data 1 and 2. The composite sensor camera 4 is a camera for capturing two types of image data with the same field of view and includes a light splitting element 5 such as a prism or a half mirror, two lenses 6, a high-spatial-resolution, low-temporal-resolution camera 7 and a low-spatial-resolution, high-temporal-resolution camera 8. The light splitting element 5 partially reflects incoming light 9. The two lenses 6 condense the light that has been reflected by the light splitting element 5 and the light that has been transmitted by the light splitting element 5, respectively. The high-spatial-resolution, low-temporal-resolution camera 7 produces an image based on the light that has been condensed by the one lens 6, thereby outputting the image data 1. On the other hand, the low-spatial-resolution, high-temporal-resolution camera 8 produces an image based on the light that has been condensed by the other lens 6, thereby outputting the image data 2.                Patent Document No. 1: Japanese Patent Application Laid-Open Publication No. 2005-318548 (FIGS. 2, 3 and 5)        